Papers with quantized models
Representation Collapse in Machine Translation Through the Lens of Angular Dispersion (2026.findings-eacl)
Copied to clipboard
| Challenge: | Effective training of Transformer models for sequential language tasks is difficult due to various forms of collapse of the internal representations learned. |
| Approach: | They propose to use angular dispersion to analyze representation collapse at different levels of discrete and continuous transformers throughout training. |
| Outcome: | The proposed method mitigates collapse and improves translation quality. |
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a low-bit training solution that requires substantial training resources. |
| Approach: | They propose an algorithm that reduces memory consumption by low-bit representations with minimal accuracy loss. |
| Outcome: | EfficientQAT achieves 2-bit Llama-2-70B model on single GPU in 41 hours . compared to previous methods, it obtains model with less than 3 points accuracy degradation . |
A Frustratingly Easy Post-Training Quantization Scheme for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Efficient inference is crucial for hyper-scale AI models, including large language models, as their parameter count continues to increase for enhanced performance. |
| Approach: | They propose a quantization scheme that fully utilizes the Transformer structure used in large language models to minimize the frequency of DRAM access while exploiting the parallelism of operations. |
| Outcome: | The proposed method minimizes the frequency of DRAM access while exploiting the parallelism of operations through a dense matrix format. |
ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Weight-only quantization reduces model size but suffers from performance degradation at lower bit widths. |
| Approach: | They propose a weight-only quantization paradigm that clusters weight matrices into codebooks and finetunes them block-by-block. |
| Outcome: | The proposed paradigm outperforms quantization methods and fine tunes LLMs to 1-bit compression and fine tuning. |
“Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization (2025.acl-long)
Copied to clipboard
| Challenge: | despite popularity of large language model quantization, there are significant accuracy-performance trade-offs associated with quantization formats. |
| Approach: | They evaluate popular quantization formats across academic benchmarks and real-world tasks . they also examine the difference in text generated by quantized models versus their uncompressed counterparts . |
| Outcome: | The proposed format is lossless across all model scales and incurs low accuracy degradation when properly tuned. |
Quantized Can Still Be Calibrated: A Unified Framework to Calibration in Quantized Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to quantify uncertainty of large language models (LLMs) but their influence on uncertainty calibration remains unexplored. |
| Approach: | They propose an analytic method to estimate the upper bound of calibration error (UBCE) for quantized LLMs and propose a method to recover calibration errors through soft-prompt tuning. |
| Outcome: | The proposed method improves the calibration accuracy of quantized models on multiple datasets and LLMs. |
Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Quantization enables efficient deployment of large language models in resource-constrained environments . but impact on truthfulness remains largely unexplored . |
| Approach: | They propose a framework to assess the truthfulness of quantized large language models . they find quantized models retain internally truthful representations but produce false outputs . |
| Outcome: | The framework assesses the truthfulness of quantized models across three dimensions . it finds that quantized model models retain internally truthful representations but are more susceptible to false outputs . |